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{SI) Abstract 

A method for detecting malignancy-associated changes. A sample of cells is obtained and stained to identify the nuclear DNA 
material. The sample is imaged with a digital microscope. Objects of interest are identified in the sample of cells based on the intensity of 
the pixels that comprise the object versus the average intensity of all pixels in the slide image. An exact edge is located for each object 
and variations in the illumination intensity of the microscope are compensated for. A computer system calculates feature values for each 
object and, based on the value of the features, a determination is made whether the cell exhibits malignancy-associated changes or not. 
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METHOD AND APPARATUS FOR AUTOMATICALLY DETECTING 
MALIGNANCY- ASSOCIATED CHANGES 
Related Applications 
The present application is a continuation-in-part of our previous application 
5 Serial No. 08/425,257 filed April 17, 1995, which was a continuation of 08/182,453 
filed January 10, 1994, which was a continuation-in-part of 07/961,596 filed 
October 14, 1992, the disclosures of which are incorporated by reference. The benefit 
of the filing dates of the previous applications are claimed under 35 U.S.C. § 120. 

Field of the Invention 

10 The present invention relates to image cytometry systems in general, and m 

particular to automated systems for detecting malignancy-associated changes in cell 
nuclei. 

Background of the Invention 
The most common method of diagnosing cancer in patients is by obtaining a 
1 5 sample of the suspect tissue and examining it under a microscope for the presence of 
obviously malignant cells. While this process is relatively easy when the location of 
the suspect tissue is known, it is not so easy when there is no readily identifiable 
tumor or pre-cancerous lesion. For example, to detect the presence of lung cancer 
from a spurum sample requires one or more relatively rare cancer cells to be present in 
20 the sample. Therefore patients having lung cancer may not be diagnosed properly if 
the sample does not accurately reflect the conditions of the lung. 

Malignancy-associated changes (MACs) are subtle changes that are known to 
take place in the nuclei of apparently normal cells found near cancer tissue. In 
addition, MACs have been detected in tissue found near pre-cancerous lesions. 
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Because the cells exhibiting MACs are more" numerous than the malignant cells, 
NlACs offer an additional way of diagnosing the presence of cancer, espee.ally >n 
cases where no cancerous cells can be located. 

Despite the ability of researchers to delect MACs in patients known to have 
cancer or a pre-cancerous condition, MACs have not yet achieved wide acceptance as 
a screening tool to determine whether a patient has or will develop cancer. 
Traditionally, MACs have been detected by carefully selecting a cell sample from a 
,ocation near a tumor or precancerous lesion and viewing the cells under relauvely 
high magnification. However, it is believed that the malignancy-assoc.ated changes 
tit take place in the cells are too subtle to be reliably detected by a human 
pathologist working with conventional microscopic equipment, especially when the 
pathologist does not know beforehand if the patient has cancer or not. For example a 
Llignancy-associated change may be indicated by the distribution of DNA wuh.n the 
nucleus coupled with slight variations in the shape of the nucleus edge. However, 
nuclei from normal cells may exhibit similar types of changes but not to the degree 
lh at would signify a MAC. Because human operators cannot easily V*™* ™* 
subtle cell changes, it is difficult to determine which cells exhrbu MAC, 
Furthermore, the changes which indicate a MAC may vary between d.fferent types of 
cancer, thereby increasing the difficulty of detecting them. 

Summary o f the. Invention 
The present invention is a system for automatically detecting malignancy- 
associated changes in cell samples. The system includes a digital microscope havtng a 
CCD camera .hat is controlled by and interfaced with a computer systenv Image 
captured by the digital microscope are stored in an image processmg board and 
.J manipulated by the computer system to detect the presence of malignancy-assocta.ed 
In es (MACs). At the present state of the an. i, is believed tha, any 
MACs reouires images to be captured at a *gh spatial resolution, a htgh photometn 
resolutron, tha, a,, information coming from the nucleus ,s tn focu, th .a 
information belongs to the nucleus (rather than some background . and tha, here 
30 an accurate and reproducible segmentation of the nucleus and nuclear matenal Each 
of these steps is described in detail below. 

To detect the malignancy-associated changes, a cell sample ,s obtamed and 
Sl ai„ed ,0 identify the nuc.ea, matenal of the cells and is imaged by the mtcroscope 
The statn is stoichiometric and specific ,0 DNA on!y. The computer system . 
35 analyzes the tmage to compute a histogram of all pixels compns.ng the tm.ge. F.rst, 
an Itensity threshold is set , ha, divides the background pi*., fi-om those compns.ng 
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the objects in the image. All pixels having an intensity value less than the threshold 
are identified as possible objects of interest while those having an intensity value 
greater than the threshold are identified as background and are ignored. 

For each object located, the computer system calculates the area, shape and 

5 optical density of the object. Those objects that could not possibly be cell nuclei are 
ignored. Next, the image is decalibrated, i.e., corrected by subtracting an empty 
frame captured before the scanning of the slide from the current frame and adding 
back an offset value equal to the average background light level. This process 
corrects for any shading of the system, uneven illumination, and other imperfections 

10 of the image acquisition system. Following decalibration, the images of all remaining 
objects must be captured in a more precise focus. This is achieved by moving the 
microscope in the stage z-direction in multiple focal planes around the approximate 
frame focus. For each surviving object a contrast function (a texture feature) is 
calculated. The contrast function has a peak value at the exact focus of the object. 

15 Only the image at the highest contrast value is retained in the computer memory and 
any object which did not reach such a peak value is also discarded from further 
considerations. 

Each remaining in-focus object on the image is further compensated for local 
absorbency of the materials surrounding the object. This is a local decalibration which 

20 is similar to that described for the frame decalibration described above, except that 
only a small subset of pixels having an area equal to the area of a square into which 
the object will fit is corrected using an equivalent square of the empty frame. 

After all images are corrected with the local decalibration procedure, the edge 
of the object is calculated, i.e., the boundary which determines which pixels in the 

25 square belong to the object and which belong to the background. The edge 
determination is achieved by the edge-relocation algorithm. In this process, the edge 
of the original mask of the first contoured frame of each surviving object is dilated for 
several pixels inward and outward. For every pixel in this frame a gradient value is 
calculated, i.e., the sum and difference between all neighbor pixels touching the pixel 

30 in question. "Then the lowest gradient value pixel is removed from the rim, subject to 
the condition that the rim is not ruptured. The process continues until such time as a 
single pixel rim remains. To ensure that the proper edge of an object is located, this 
edge may be again dilated as before, and the process repeated until such time as the 
new edge is identical to the previous edge. In this way the edge is calculated along 

35 the highest local gradient. 
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The computer system .hen calculates a set of feature values for each object. 
For some elLo,! .ions the edge along the highest gradient values correced by 
k d lithe edee by one or more pixels or eroding the edge by one or more 

IX malignancyassociated changes. Based on ,he number o o^ts four. ^ 

™le that appear to have malignancy-assoc.ated changes and/or 
I a c",ed score, a deter— can be made whether the pat.ent from 
^".he cell sample was obtained is heahhy or harbors a maUgnan, growth. 

pri»f r>AcrH ptinn of 'hp Drawings 
The foregoing aspects and many of the attendant advantages of this mven.ron 
, wi„ belt mo 8 re ,U appreciated as the same becomes * 
reference to the following detailed desenptton. when taken ,n conjunct 

^^Z^^Z^ - the MAC detection system according to the 
, 0 nou^s a«C are a series of flow charts showing the steps performed by 

interest from the background of a slide; nrocedure used to prepare 

FIGURE 4 is a flow chart of the preferred sta.rung procedure P 
•><; a cell sample for the detection of MACs; 

FIGURES 5 and 6 are illustrations of objects !oca,ed » an rmage 

FIGURES iUustra.e how the present invention operates to locate the 

edge of an object; ri Wmer that separates 

FIGURES 8 and 9 are diagrammatic .llustrauons of a classiher P 

30 artifacts from cell nuclei and MAC nuclei from ™^ C ^* invemion t0 
FIGURE 10 is a flow chart of the steps performed by the present in 

„ hether a patient is normal or abnormal based on the presence of MACs. 

determine whether a patiem ib uu L^m^nt 

automaticany 

- aesenbed above, « ^ ™ ^ -h . 
3 5 detecting malignancy-associated changes (MALs) 
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a patient. From the presence or absence of MACs, a determination can be made 
whether the patient has a malignant cancer. 

A block diagram of the MAC detection system according to the present 
invention is shown in FIGURE 1. The system 10 includes a digital microscope 12 
that is controlled by and interfaced with a computer system 30. The microscope 12 
preferably has a digital CCD camera 14 employing a scientific CCD having square 
pixels of approximately 0.3 \xm by 0.3 jam size. The scientific CCD has a 100% fill 
factor and at least a 256 gray level resolution. The CCD camera is preferably 
mounted in the primary image plane of a planar objective lens 22 of the 
0 microscope 12. 

A cell sample is placed on a motorized stage 20 of the microscope whose 
position is controlled by the computer system 30. The motorized stage preferably has 
an automatic slide loader so that the process of analyzing slides can be completely 
automated. 

5 A stable light source 18, preferably with feedback control, illuminates the cell 

sample while an image of the slide is being captured by the CCD camera. The lens 22 
placed between the sample 16 and the CCD camera 14 is preferably a 20x/0.75 
objective that provides a depth of field in the range of 1-2 jam that yields a distonion- 
free image. In the present embodiment of the invention, the digital CCD camera 14 

0 used is the Microimager™ produced by Xillix Technologies Corp. of Richmond, 
B.C., Canada. 

The images produced by the CCD camera are received by an image processing 
board 32 that serves as the interface between the digital camera 14 and the computer 
system 30. The digital images are stored in the image processing board and 
5 manipulated to facilitate the detection of MACs. The image processing board creates 
a set of analog video signals from the digital image and feeds the video signals to an 
image monitor 36 in order to display an image of the objects viewed by the 
microscope. 

The computer system 30 also includes one or more input devices 38, such as a 
10 keyboard and mouse, as well as one or more peripherals 42, such as a mass digital 
storage device, a modem or a network card for communicating with a remotely 
located computer, and a monitor 40. 

FIGURES 2A-2C show the steps performed by the system of the present 
invention to determine whether a sample exhibits MACs or not. Beginning with a 
15 step 50, a cell sample is obtained. Cells may be obtained by any number of 
conventional methods such as biopsy, scraping, etc. The cells are affixed to a slide 
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and stained us,ng a modified Feu.gen procedure* a step 52 *a, identifies the nuc^r 
DNA in .he sample. The details of .he staining procedure are shown m FIGURE 4 

and described in detail below. 

A. s,ep 54, an image of a frame from .he slide is captured by .he CCD camera 
and is transferred into ,he image processor, .n .h,s process, ,he CCD sensor wnh.n 
.he camera is cleared and a shutter of .he camera is opened for a fixed penod tha. ,s 
dependent on the intensity of the light source 18. After the image ,s opumrzed 
according .0 .he s.eps described below, .he s.age .hen moves <o a new pos.t.on on ,he 
de such tha. another image of .he new frame can be cap.ured by .he camera and 
transferred into .he compu.er memory. Because .he cell sample on .he s hde occup.e 
a much greater area than the area viewed by the microscope, a number of sl.de .mages 
are used .o determine whe.her the sample is MAC- P osi.ive or nega.ive. The posn.on 
of each cap.ured image on .he slide is recorded in ,he compu.er sys.em so .ha. .he 
objecs of interest in .he image can be found on .he slide ,f desued. 
« Once an image from the slide is cap.ur«d by the CCD camera and s.ored n .he 

image processing board. ,he compu.er sys.em de,ermines whe.her ,he tmage pro uc d 
by .he CCD camera is devoid of objecs. This is performed by scannmg the d.gua 
image for dark pixels. If .he number of dark pixels, i.e., those j*eU havmg an 
intensny of the background ,n.ens,.y minus a predetermined offset value, ,s fewer , „ 
,0 a predetermined minimum, .he compu.er system assumes that the ,m.ge „ Wank and 
Z microscope stage is moved .0 a new position a. ,ep 60 and a new ,mage ,s 

^'IfH il'ge is no. blank, then the computer sys.em auempts to globally focus 
the image. In genera,, when the image is in focus, the objects 
1S have a maximum darkness. Therefore, for focus determinate the haght of the ge 
justed and a new image is captured. The darkness of .he o b) ec. ptxels 
eterled and .he process repeats until the average darkness of , e ptxe * » h 
image is a maximum. A. .his poin, .he compu.er sys.em assumes tha. global focus 

has been obtained. . ^ 

After perfor^ng the rough, global focus a, step 62, the computer s stem 
compu,es a his,ogram of all pixels. As shown in FIGURE 3. a his.ogram ,s a plot of 
he number of pixels a, each intensity level. In .he Mic,oimager™-based rrucroscop 
sy , m each pixel can have an in.ensi.y rangmg from 0 (maximum darkne s) _.o 
<— brightness,. The histogram typtcally contains a - ^ h = 
35 represents the average intensity of the background pixels. A ^ 
represents .he average intensity of .he pixels ,ha. compnse the ob.ects By ca.culat.ng 
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a threshold 94 that lies between the peaks 90 and 92, it is possible to crudely separate 
the objects of interest in the image from the background. 

Returning to FIGURE 2B, the computer system computes the threshold that 
separates objects in the image from the background at step 68. At a step 72, alt pixels 
5 in the cell image having an intensity less than the threshold value are identified. The 
results of step 72 are shown in FIGURE 5. The frame image 200 contains numerous 
objects of interest 202, 204, 206 .. . 226. Some of these objects are cell nuclei, which 
will be analyzed for the presence of MACs, while other objects are artifacts such as 
debris, din particles, white blood cells, etc., and should be removed from the cell 
10 image. 

Returning to FIGURE 2B, once the objects in the image have been identified, 
the computer system calculates the area, shape (sphericity) and optical density of each 
object according to formulas that are described in further detail below. At a step 76, 
the computer system removes from memory any objects that cannot be cell nuclei. In 
1 5 the present embodiment of the invention those objects that are not possibly cell nuclei 
are identified as having an area greater than 2,000 urn 2 , an optical density less than 1 c 
(i.e., less that 1/2 of the overall chromosome count of a normal individual) or a shape 
or sphericity greater than 4. 

The results of step 76 are shown in FIGURE 6 where only a few of the 
20 previously identified objects of interest remain. Each of the remaining objects is more 
likely to be a cell nuclei that is to be examined for a malignancy-associated change. 

Again returning to FIGURE 2B, after removing each of the objects that could 
not be a cell nucleus, the computer system determines whether there are any objects 
remaining by scanning for dark pixels at step 78. If no objects remain, the computer 
25 system returns to step 54, a new image on the slide is captured and steps 54-76 are 
repeated. 

If there are objects remaining in the image after the first attempt at removing 
artifacts at step 76, the computer system then compensates the image for variations in 
illumination intensity at step 80. To do this, the computer system recalls a calibration 

30 image that was obtained by scanning in a blank slide for the same exposure time that 
was used for the image of the cells under consideration. The computer system then 
begins a pixel-by-pixel subtraction of the intensity values of the pixels in the 
calibration image obtained from the blank slide from the corresponding pixels found in 
the image obtained from the cell sample. The computer system then adds a value 

35 equal to the average illumination of the pixels in the calibration image obtained from 



WO 97/43732 



-8- 



PCT/CA97/00301 



10 



.1- , n8 Thf result of the addition illuminates 
the blank slide to each pixel of the cell .mage. The result 

(FIGURE 2C). The optimum focus « obtamed " °^ move , 

l„ d maxtmum darkness. The computer system .he, fo,e -se S sequence of 

nredenned amount above -he 8 ,o ^ _ an of , he 

cor^KomtheposUionin^ 

darkness and occupy a — area If the opt, ^ ^ 

predetermined numb, of s,a 8 e pos.uons , b o >e ^ ^ ^ ^ ^ 

TI Sec, of Lest uU^eW recorded , ,.s best poss.b e ^ 
At a step 84, the computer system determtnes 
„ , h e ce„ ima 6 e were found. If no,. ,he ^""^Lw ™* * 
FIGURE 2A whereby ,he slide is moved ,0 another pos.t.on 

CaP,Ure 0„ce an .ma 8 e of the ob.c, has ^^^oZXZ 

25 computer system analyzes a number of p.xels w ^ ^ 

.hanmeobjectbytwopixeisonalistdes. An exam of sue 
sh0 »n in FIGURE 6. The computer system £ J^,,, 

30 SXTr - each pixe, - -^Jir^ t 

the box that encloses the object. madCj the 

Once the compensate for — J ^ achlatnin, object in the 
35 compu.er system then determines a more prease edg 
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eel! image at step 86. The steps required to compute the edge are discussed in further 
detail below. 

Having compensated for local absorbency and located the precise edge of the 
object, the computer system calculates a set of features for each remaining object at a 
5 step 87. These feature values are used to further separate artifacts from cell nuclei as 
well as to identify nuclei exhibiting MACs. The details of the feature calculation are 
described below. 

At a step 88, the computer system runs a classifier that compares the feature 
values calculated for each object and determines whether the object is an artifact and, 
1 0 if not, whether the object is a nucleus that exhibits MACs. 

At a step 90, the pseudo-focus digital image, the feature calculations and the 
results of the classifier for each in-focus object are stored in the computer's memory. 

Finally, at a step 92, the computer system determines whether further scans of 
the slide are required. As indicated above, because the size of each cell image is much 
1 5 less than the size of the entire slide, a number of cell images are captured to ensure 
that the slide has been adequately analyzed. Once a sufficient number of cell images 
have been analyzed, processing stops at step 94. Alternatively, if further scans are 
required, the computer system loops back to step 54 and a new image of the cell 
sample is captured. 

20 As indicated above, before the sample can be imaged by the digital 

microscope, the sample is stained to identify the nuclear material. 

FIGURE 4 is a flow chart of the steps used to stain the cell samples. 
Beginning at a step 100, the cell sample is placed on a slide, air dried and then soaked 
in a 50% glycerol solution for four minutes. The cell is then washed in distilled water 

25 for two minutes at a step 102. At a step 104, the sample is bathed in a 50% ethanol 
solution for two minutes and again washed with distilled water for two minutes at a 
step 106. The sample is then soaked in a Bohm- Springer solution for 30 minutes at a 
step 108 followed by washing with distilled water for one minute at a step 110. At 
step 112, the sample is soaked in a 5N HC1 solution for 45 minutes and rinsed with 

30 distilled water for one minute at a step 114. The sample is then stained in a thionine 
stain for 60 minutes at a step 1 16 and rinsed with distilled water for one minute at a 
step 118. 

At step 120, the sample is soaked in a bisulfite solution for six minutes 
followed by a rinse for one minute with distilled water at a step 122. Next, the sample 
35 is dehydrated in solutions of 50%, 75% and 100% ethanol for approximately 10 
seconds each at a step 124. The sample is then soaked in a final bath of xylene for 
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one minute a. a .up 126 before a cove, slip is applied at a step 128. After the cell 
sample has been prepared, it is ready to be imaged by the digital mrcroscope and 

analyzed as described above. 

FIGURES 7A-7F illustrate the manner in which the present mvenuon 
calculates the precise edge of an object. As shown in FIGURE 7 an object 230 ,s 
comprised of those pixels having an intensity value less than the background/object 
threshold which is calculated from the histogram and described above. In order to 
calculate the precise edge, the pixels lying at the origina. edge of the object are d.lated 
to form a new edge region 242. A second band of pixels lying inside the original edge 
are also selected to form a second edge region 244. The computer system then 
assumes that the true edge is somewhere within the annular ring bounded by the edge 
regions 242 and 244. In the presently preferred embodiment of the mvenuon, the 
annular nng has a width of approximately ten pixels. To determine the edge, the 
computer calculates a gradient for each pixel contained in the annular nng. The 
gradient for each pixel is defined as the sum of the differences in intensuy between 
each pixel and its surrounding eight neighbors. Those pixels having neighbors w, h 
similar intensity levels will have a low gradient while those pixels at the edge of the 
object will have a high gradient. ' Uo 

Once the gradients have been calculated for each pixel m the annular nng. the 
, computer system divides the range of gradients into multiple thresholds and begms 
removing pixels having lower gradient va.ues from the ring. To remove the p.xel 
the computer scans the object under consideration ,„ a raster fashron. As shown n 
F.GUKE7C. Ihe raster scan begins a, a point A and continues ,0 the ngh. 
reaching a point B. During the first scan, only pixels on the outside edge, ,.e.. prxels 
5 on the edge region 242, are removed. The computer system then scans ,n t 
opposite direction by starting, for example, a, point D and contmumg upwards 0 
point B re.unung in a raster fashion while only removing pixels on the rns.de edge 
egion244 of the annular ring. The computer system then scans ,n another 
orthogonal direction-for example, starting a. point C and continuing m the dtreouon 
of po ntD ir, a raster fashion, this time only removing orxels on the ou.s,de edg 
regln242. This process continues until no more pixels a, tha, grad.en, threshold 

value can be removed. . 

Pixels are removed from the annular ring subject to the condmons that no 
pixel can be removed that would break the chain of pixels around the annular nng. 
Furthermore, adjacent pixels cannot be removed dunng the same pass o P x 
removal Once all the pixels are removed having a gradrent that ,s less than or enua, 
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to the first gradient threshold, the threshold is increased and the process starts over. 
As shown in FIGURE 7D, the pixel-by-pixel removal process continues until a single 
chain of pixels 240' encircles the object in question. 

After locating the precise edge of an object, it is necessary to determine 
5 whether those pixels that comprise the edge should be included in the object. To do 
this, the intensity of each pixel that comprises the newly found edge is compared with 
its eight neighbors. As shown in FIGURE 7E, for example, the intensity of a 
pixel 246 is compared with its eight surrounding pixels. If the intensity of pixel 246 is 
less than the intensity of pixel 250, then the pixel 246 is removed from the pixel chain 
0 as it belongs to the background. To complete the chain, pixels 248 and 252 are added 
so that the edge is not broken as shown in FIGURE 7F. After completing the edge 
relocation algorithm and determining whether each pixel should be included in the 
object of interest, the system is ready to compute the feature values for the object. 

Once the features have been calculated for each in-focus object, the computer 
5 system must make a determination whether the object is a ceil nucleus that should be 
analyzed for malignancy-associated changes or is an artifact that should be ignored. 
As discussed above, the system removes obvious artifacts based on their area, shape 
(sphericity) and optical density. However, other artifacts may be more difficult for the 
computer to recognize. To further remove artifacts, the computer system uses a 
20 classifier that interprets the values of the features calculated for the object. 

As shown in FIGURE 8, a classifier 290 is a computer program that analyzes 
an object based on its feature values. To construct the classifier two databases are 
used. The first database 275 contains feature values of objects that have been imaged 
by the system shown in FIGURE 1 and that have been previously identified by an 
25 expert pathologist as non-nuclei, i.e., artifacts. A second database 285 contains the 
features calculated for objects that have been imaged by the system and that have been 
previously identified by an expert as cell nuclei. The data in each of these databases is 
fed into a statistical computer program which uses a stepwise linear discriminant 
function analysis to derive a discriminant function that can distinguish cell nuclei from 
30 artifacts. The classifier is then constructed as a binary decision tree based on 
thresholds and/or the linear discriminant functions. The. binary tree answers a series 
of questions based on the feature values to determine the identity of an object. 

The particular thresholds used in the binary tree are set by statisticians who 
compare histograms of feature values calculated on known objects. For example, 
35 white blood cells typically have an area less than 50^m 2 . Because the present 
invention treats a red blood cell as an artifact, the binary decision tree can contain a 
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10 



rode that compares the area of an object to the 50um 2 threshold. Objects with an 
area less than the threshold are ignored while those with an area having a greater area 
are farther analyzed to determine if they are possible MAC cells or artifacts. 

In the presently preferred embodiment of the invention, the discriminant 
functions that separate types of objects are generated by the BMDP program available 
from BMDP Statistical Software, Inc., of Los Angeles, California. Given the 
discriminant functions and the appropriate thresholds, the construction of the binary 
tree classifier is considered routine for one of ordinary skill in the art. 

Once the binary tree classifier has been developed, it can be supplied with a set 
of feature values 292 taken from an unknown object and will provide an 
indication 294 of whether the object associated with the feature data is most likely an 

artifact or a cell nucleus. 

FIGURE 9 shows how a classifier is used to determine whether a slide exhibits 
malignancy-associated changes or not. The classifier 300 is constructed using a pair 
1 5 of databases. A first database 302 contains feature values obtained from apparently 
normal cells that have been imaged by the digital microscope system shown in 
FIGURE 1 and are known to have come from healthy patients. A second 
database 304 contains feature values calculated from apparently normal cells that 
were imaged by the digital microscope system described above and were known to 
have come from abnormal (i.e., cancer) patients. Again, classifier 30.0 used in the 
presently preferred embodiment of the invention is a binary decision tree made up of 
discriminant functions and/or thresholds that can separate the two groups of cells. 
Once the classifier has been constructed, the classifier is fed with the feature 
values 306 that are obtained by imaging cells obtained from a patient whose condition 
25 is unknown. The classifier provides a determination 308 of whether the nuclei exhibit 
MACs or not. 

FIGURE 10 is a flow chart of the steps performed by the present invention to 
determine whether a patient potentially has cancer. Beginning at a step 325, the 
computer system recalls the features calculated for each in-focus nuclei on the slide. 
30 At a step 330, the computer system runs the classifier that identifies MACs based on 
these features. At a step 332, the computer system provides an indication of whether 
the nucleus in question is MAC-positive or not. If the answer to step 332 is yes, then 
an accumulator that totals the number of MAC-positive nuclei for the slide is 
increased at a step 334. At a step 336, the computer system determines whether all 
35 the nuclei for which features have been calculated have been analyzed. If not, the 
next set of features is recalled at step 338 and the process repeats itself. At a 
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siep 340, the computer system determines whether the frequency of MAC-positive 
cells on the slide exceeds a predetermined threshold. For example, in a particular 
preparation of cells (air dried, as is the practice in British Columbia, Canada) to detect 
cervical cancer, it has been determined that if the total number of MAC-positive 
5 epithelial cells divided by the total number of epithelial cells analyzed exceeds 0.45 per 
slide, then there is an 85% chance that the patient has or will develop cancer. If the 
frequency of cells exhibiting MACs exceeds the threshold, the computer system can 
indicate that the patient is healthy at step 342 or likely has or will develop cancer at 
step 344. 

[0 The threshold above which it is likely that a patient exhibiting MACs has or 

will develop cancer is determined by comparing the MAC scores of a large numbers 
of patients who did develop cancer and those who did not. As will be appreciated by 
those skilled in the art, the particular threshold used will depend on the type of cancer 
to be detected, the equipment used to image the ceils, etc. 
15 The MAC detection system of the present invention can also be used to 

determine the efficacy of cancer treatment. For example, patients who have had a 
portion of a lung removed as a treatment for lung cancer can be asked to provide a 
sample of apparently normal cells taken from the remaining lung tissue. If a strong 
MAC presence is detected, there is a high probability that the cancer will return. 
20 Conversely, the inventors have found that the number of MAC cells decreases when a 
cancer treatment is effective. 

As described above, the ability of the present invention to detect malignancy- 
associated changes depends on the values of the features computed. The following is 
a list of the features that is currently calculated for each in-focus object. 
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1.2 Coordinate Systems, Jargon and Notation 

Each image is a rectangular array of square pixels that contains within it the 
image of an (irregularly shaped) object, surrounded by background. Each pixel P u is 
an integer representing the photometric value (gray scale) of a corresponding small 

5 segment of the image, and may range from 0 (completely opaque) to 255 (completely 
transparent). The image rectangle is larger than the smallest rectangle that can 
completely contain the object by at least two rows, top and bottom, and two columns 
left and right, ensuring that background exists all around the object. The rectangular 
image is a matrix of pixels, P iJt spanning / = 1, L columns and; = 1, M rows and with 

10 the upper left-hand pixel as the coordinate system origin, ;'=;'= 1 . 

The region of the image that is the object is denoted by its characteristic 
function, O; this is also sometimes called the "object mask" or, simply, the "mask." 
For some features, it makes sense to dilate the object mask by one pixel all around the 
object; this mask is denoted Q\ Similarly, an eroded mask is denoted fi\ The object 

1 5 mask is a binary function: 

where 

_ Jl'/ if, J) £ object 
'•' ~ \0if « object 
and where u (iJ) e object" means pixels at coordinates: (i, j) are part of the object, 
20 and "(ij) « object" means pixels at coordinates: (i, j) are not part of the object. 

IJ Morphological Features 

Morphological features estimate the image area, shape, and boundary 
variations of the object. 

n.l area 

25 The area, A, is defined as the total number of pixels belonging to the object, as 

defined by the mask, Q: 

L M 

area = A = Y*HL®-ij 



(2) 



where ij and Q are defined in Section 1.2 above. 
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n.2 x_centroid, y_centroid 

The x_centroid and y_centroid are the coordinates of the geometrical center of 
the object, defined with respect to the image origin (upper-left hand corner): 

L M 

x_ centroid = J ^ ^ (3) 

L M 

z I /-n, , 

3 y_ centroid - — (4) 

where / andy are the image pixel coordinates and Q is the object mask, as defined in 
Section 1.2 above, and A is the object area. 

tl.3 mean_radius, max_radius 

The mean__radius and max__radius features are the mean and maximum values 
10 of the length of the object's radial vectors from the object centroid to. its 8 connected 
edge pixels: 

mean radius = r = — ($\ 
N V ' 

max_ radius - max(r k ') (6) 
where r k is the h* radial vector, and N is the number of 8 connected pixels on the 
15 object edge. 

II.4 var_radius 

The var_radius feature is the variance of length of the object's radius vectors, 
as defined in Section II. 3. 

var radius - — rn\ 

20 where r k is the radius vector, f is the mean_radius, and N is the number of 8 
connected edge pixels. 
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11.5 sphericity 

• The sphericity feature is a shape measure, calculated as a ratio of the radii of 
two circles centered at the object centroid (defined in Section II. 2 above). One circle 
is the largest circle that is fully inscribed inside the object perimeter, corresponding to 
5 the absolute minimum length of the object's radial vectors. The other circle is the 
minimum circle that completely circumscribes the object's perimeter, corresponding 
to the absolute maximum length of the object's radial vectors. The maximum 
sphericity value: 1 is given for a circular object: 

. min_radius min(r k ) 

sphericity = — = - - (8) 

max_ radius max( ) 

1 0 where r k is the radius vector. 
n.6 eccentricity 

The eccentricity feature is a shape function calculated as the square root of the 
ratio of maximal and minimal eigenvalues of the second central moment matrix of the 
object's characteristic function, £1: 



1 5 eccentricity - I — (9) 




where and X 2 are the maximal and minimal eigenvalues, respectively, and the 
characteristic function, H, as given by Equation 1. The second central moment matrix 
is calculated as: 



^ mom* nil crQjsmom*nl7 
' crossmomtntl ¥ momtntl 



(10) 
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L M 
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Eccentricity may be interpreted as the ratio of the major axis to minor axis of the "best 
fit" ellipse which describes the object, and gives the minimal value 1 for circles. 

5 11.7 inertia_shape 

The inertia_shape feature is a measure of the "roundness" of an object 
calculated as the moment of inertia of the object mask, normalized by the area 
squared, to give the minimal value 1 for circles: 

L M , 

2*E z*?,n,.,. 

inertia _ shape = — ^ q 

10 where Rjj is the distance of the pixel, Pjj t to the object centroid (defined in 
Section II. 2), and A is the object area, and CI is the mask defined by Equation 1 . 

n.8 compactness 

The compactness feature is another measure of the object's "roundness. " It is 
calculated as the perimeter squared divided by the object area, giving the minimal 
1 5 value 1 for circles: 

P 2 

compactness*— (12) 

where P is the object perimeter and A is the object area. Perimeter is calculated from 
boundary pixels (which are themselves 8 connected) by considering their 4 connected 
neighborhood: 
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P = N. + V2// 3 + 2N, 



(13) 



where Nj is the number of pixels on the edge with 1 non-object neighbor, N 2 is the 
number of pixels on the edge with 2 non-object neighbors, and N 3 is the number of 
pixels on the edge with 3 non-object neighbors. 



II.9 ccII_orient 

The cell_orient feature represents the object orientation measured as a 
deflection of the main axis of the object from the v direction: 



180 

cell orient = 

~ 7T 



n 

— + arctan 



(14) 



cross momsnt7 

where v and xv , are the second central moments of the characteristic 

function CI defined by Equation 1 above , and X, is the maximal eigenvalue of the 
second central moment matrix of that function (see Section II.6 above). The main axis 
of the object is defined by the eigenvector corresponding to the maximal eigenvalue. 
A geometrical interpretation of the cell^orient is that it is the angle (measured in a 

clockwise sense) between thejy axis and the "best fit" ellipse major axis. 

For slides of cell suspensions, this feature should be meaningless, as there 

should not be any a priori preferred cellular orientation. For histological sections, and 

possibly smears, this feature may have value. In smears, for example, debris may be 

preferentially elongated along the slide long axis. 



n.10 elongation 

Features in Sections 11.10 to 11.13 are calculated by sweeping the radius vector 
(from the object centroid, as defined in Section II.2, to object perimeter) through 128 
discrete equal steps (i.e., an angle of 271/128 per step), starting at the top left-most 
object edge pixel, and sweeping in a clockwise direction. The function is interpolated 
from an average of the object edge pixel locations at each of the 128 angles. 

The elongation feature is another measure of the extent of the object along the 
principal direction (corresponding to the major axis) versus the direction normal to it. 
These lengths are estimated using Fourier Transform coefficients of the radial function 
of the object: 
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a 0 +2. 



elongation = 



2 u 2 
a +6 

2 2 



2 u 2 
a 0 -2.\a +b 



(15) 



2 2 

where a 1( A, are Fourier Transform coefficients of the radial function of the object, 
r(9), defined by: 



r(6) = ^ + £>. coinB) + f,b n sin(*0) 

2 „M n=) 



(16) 



ILll freqJo\v_fTt 

The freq_low_flft gives an estimate of coarse boundary variation, measured as 
the energy of the lower harmonics of the Fourier spectrum of the object's radial 
function (from 3rd to 1 1th harmonics): 



freq_ low_ ffi=T. (a 2 n + b 2 n ) (17) 



n=2 

10 where a n \ are Fourier Transform coefficients of the radial function, defined in 
Equation 16. 

11.12 freq_high fft 

The freq_high_fft gives an estimate of the fine boundary variation, measured 
as the energy of the high frequency Fourier spectrum (from 12th to 32nd harmonics) 
1 5 of the object's radial function; 

32 ( 2 2^ 

freqhigh fft = T! a +6 0 8 ) 

where a n ,b n are Fourier Transform coefficients of the m harmonic, defined by 
Equation 16. 
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D.13 harmon01_fft, harmon32_fft 

The harmonOlJft, ... hanmon32_ffi features are estimates of boundary 
variation, calculated as the magnitude of the Fourier Transform coefficients of the 
object radial function for each harmonic 1 - 32: 



J 2 2 
a +b (19) 
n n 

where a„,b„ are Fourier Transform coefficients of the » lh harmonic, defined by 
Equation 16. 

Ill Photometric Features 

Photometric features give estimations of absolute intensity and optical density 
1 0 levels of the object, as well as their distribution characteristics. 

III.l DNA_Amount 

DNA_Amount is the "raw" (unformalized) measure of the integrated optical 
density of the object, defined by a once dilated mask, Q + : 

L M 

DNA Amount =IZ OD^Ofj (20) 
/=l/=l 

15 where the once dilated mask, Q + is def.ned in Section 1.2 and OD is the optical 
density, calculated according to 112], 

ODjj = log x o J B ~ l °SlO J i.j (2 1 ) 

where l B is the intensity of the local background, and I tJ is the intensity of the ij th 
pixel. 

20 1H.2 DNAJndex 

DNAJndex is the normalized measure of the integrated optical density of the 

object: 

DNA Amount 



DNA Index = 



iod norm 



(22) 



where wd norm is the mean value of the DNA amount for a particular object population 
25 from the slide (e.g., leukocytes). 
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JU.3 var_intcnsity, mcan_intensity 

The varjntensity and meanjntensity features are the variance and mean of the 
intensity function of the object, /, defined by the mask, H: 

var_ intensity = — - — j (23) 

5 where A is the object area, fi is the object mask defined in Equation 1, and / is given 

by: 

L M 

j = i^M (24) 

A 

I is the "raw" (unnonmalized) mean intensity. 

mean intensity is normalized against iod norm defined in Section IH.2: 

10 mean intensity = J^™™1 ( 2 S) 

" .100 

111.4 OD_maximum 

OD_maxjmum is the largest value of the optical density of the object, 
normalized to iod as defined in Section III. 2 above: 



OD_ maximum = max(ODjj) 



f 100 ^ 



lod normJ 



(26) 



15 FEL5 ODvariance 

OD_variance is the normalized variance (second moment) of optical density 
function of the object: 

LZmjClij-OD) 2 

OD variance = llkd __ (27) 

{A-\)OD 

where Q is the object mask as defined in Section 1.2, OD is the mean value of the 
20 optical density of the object: 
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0D = 



( l m "\ 

I I ODfjClij 



and A is the object area (total number of pixels). The variance is divided by the 
square of the mean optical density in order to make the measurement independent of 
the staining intensity of the cell. 

H3.6 OD_skewness 

The OD skewness feature is the normalized third moment of the optical 
density function of the object: 



OD skewness = 



L M 

ZTLiODijCltj-OD)* 

( I M , 

(/i-i) zz(OAA>-^r 

V/=iy=i 



(28) 



where CI is the object mask as defined in Section 1.2, OD is the mean value of the 
optical density of the object and A is the object area (total number of pixels). 



IH.7 OD_kurtosis 

OD_kurtosis is the normalized fourth moment of the optical density function 
of the object . 



OD kiirtosis - 



L M 4 

ZZCODijClij-OD) 4 

( L M 7 

(A-lAzZfODuClu-OD) 2 



(29) 



where Q is the object mask as defined in Section 1.2, OD is the mean value of the 
optical density of the object , and A is the object area. 
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IV Discrete Texture Features 

The discrete texture features are based on segmentation of the object into 
regions of low, medium and high optical density. This segmentation of the object into 
low, medium and high density regions is based on two thresholds: optical density high 
5 threshold and optical density medium threshold. These thresholds are scaled to the 
sample's iod n0 rni value, based on the DNA amount of a particular subset of objects 
(e.g., lymphocytes), as described in Section III. 2 above. 

By default, these thresholds have been selected such that the condensed 
chromatin in leukocytes is high optical density material. The second threshold is 
1 0 located half way between the high threshold and zero. 

The default settings from which these thresholds are calculated are stored in 
the computer as: 



A mcd - is the area of the pixels having an optical density between 18 and 36 and Al° w 

is the area of the pixels having an optical density greater than 36. Together the areas 
A high A mcd an d A Iow 

sum to the total area of the object. The actual thresholds used 
are these parameters, divided by 100, and multiplied by the factor iod norm / 100. 



medium-, and high-optical density regions of the object, respectively, defined in 
analogy to Equation 1 . 

rv.l lowDNAarea, medDNAarca, hiDNAarea 

These discrete texture features represent the ratio of the area of low, medium, 
25 and high optical density regions of the object to the total object area: 



CHROMATIN HIGH THRES = 36 



15 



CHROMATIN_MEDIUM_THRES = 18 
A hi £ h is the area of the pixels having an optical density between 0 and 18, 



20 



In the following discussion, f} low , Q mcd , and Q h,sh are masks for low-, 




A 



low 



lowDNAarea = 



(30) 



L M 



A 



S 2 Clij 
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V y ci™ ed 

medDNAarea = '—^ = — j- ( 31 ) 

2 Z Q /J 
,=1;=1 

hiDNAarea = ^ = — (32) 

Z Z j 
,=iy=i 

where £~l is the object mask as defined in Equation 1, and A is the object area. 



IV.2 lowDNAamnt, medDNAamnt, hiDNAamnt 

These discrete texture features represent the total extinction ratio for low, 
medium, and high optical density regions of the object, calculated as the value of the 
integrated optical density of the low-, medium-, and high-density regions, 
respectively, divided by the total integrated optical density: 



lowDNAamnt = '-^ ( 33 > 

£ Z ODijClij 
,=iy=i 

10 medDNAamnt = ( 34) 

Z Z ODijSlij 
i=iy=i 

I Z ODijClTj 

hiDNAamnt = < 3 5) 

Z Z ODfjClij 

where Q is the object mask as defined in Equation 1, and OD is the optical density as 
defined by Equation 21. 
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IV.3 lowDNAcomp, medDNAcomp, hiDNAcomp, mhDNAcomp 

These discrete texture features are characteristic of the compactness of low-, 
medium-, high-, and combined medium- and high-density regions, respectively, 
treated as single (possibly disconnected) objects. They are calculated as the perimeter 
5 squared of each region, divided by 4k (area) of the region. 

/ plows! 

lowDNAcomp = - j— (36) 

, pined *2 

medDNAcomp = * -r (37) 

AnA med 

(P hi ) 2 

hiDNAcomp = - '— (38) 

4nA h ' 

mhDNAcomp = iPmed * Phi) hi ( 39 > 
4n(A med +A ht ) 

10 where P is the perimeter of each of the optical density regions, defined in analogy to 
Equation 13, and A is the region area, defined in analogy to Equation 2. 

rV.4 low av dst, mcd_av_dst, hi_av_dst, mh_av_dst 

These discrete texture features represent the average separation between the 
low-, medium-, high-, and combined medium- and high-density pixels from the center 
1 5 of the object, normalized by the object mean_radius. 

low av_dst = -^ 1 (40) 

A ■ mean _ radius 

L M . 
Z £ Ri.jO.Tf 

med_av_dst= '"'j" 1 (41) 

A ■ mean_ radius 

L M 

hi_av d« = -p^ (42) 

A ■ mean radius 
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L M 



i z Ki.pTf + 1 z \P?j 



L M 



mh av dsi = 



f^'""* + A hi )■ mean _ radius 



(43) 



where is defined in Section II.7 as the distance from pixel P u to the object 
centroid (defined in Section II.2), and the object mearwadius is def.ned by Equation 
5. 



10 



IV.5 lowVSmcdJDNA, lowVShighJDNA, lowVSmh_DNA 

These discrete texture features represent the average extinction ratios of the 
low- density regions, normalized by the medium-, high-, and combined medium- and 
high-average extinction values, respectively. They are calculated as the mean optical 
density of the medium-, high-, and combined medium- and high-density clusters 
divided by the mean optical density of the low density clusters. 



lowVSmed DNA = 



Z Z ODtjClTf 

<=iy=i 



A med 



Z Z OD t .jQfj 



4 lOW 



(44) 



lowVShi_DNA = 



Z Z ODijClTj 



low 



Z Z ODj jQ.'i 0 } 



(45) 



( L M 



lowVSmh DNA = 



L M 



A med + A hi 



f L M 



Jow 



Z Z ODtjSlfi 
<=iy=i 



t low 



(46) 



where OD is the region optical density defined in analogy to Equation 21, SI is the 
15 region mask, defined in analogy to Equation 1, and A is the region area, defined m 
analogy to Equation 2. 
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T\ 7 .6 low_den_obj, med_den_obj, high_den_obj 

These discrete texture features are the numbers of discrete 8-connected 
subcomponents of the objects consisting of more than one pixel of low, medium, and 
high density. 

IV.7 lovv_cntr_mass, med cntrjnass, high_cntr_mass 

These discrete texture features represent the separation between the geometric 
center of the low, medium, and high optical density clusters (treated as if they were 
single objects) and the geometric center of the whole object, normalized by its 
mean radius. 



10 low cntr mass = 



,=l , x centroid 



\ 2 / L M 

' =l J * x , v centroid 

A tow J - 



+ (mean __ radius) 
(47) 



mcd cntr mass = 



( L M 



zZ'-«;r 



A" 



■ - x centroid 



zz^r," 

y "' , v centroid 



J 



- (mean _ radius) 



(48) 



15 



hi cntr mass = 



ZZ'^ 

'" * ; " 1 , a" centroid 

A h 



\ 2 ( l M \ 

'" ] , v centroid 

j hi J — 



-s- (mean _ radius) 



(49) 



where mean_radius of the object is defined by Equation 5, the object's centroid is 
defined in Section II. 2, Q is the region mask defined in analogy to Equation 1, and A 
is the region area defined in analogy to Equation 2. 
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V Markovian Texture Features 

Markovian texture features are defined from the co-occurrence matrix, A XjJ of 
object pixels. Each element of that matrix stands for the conditional probability of the 
pixel of grey level X occurring next (via 8-connectedness) to a pixel of grey level u, 
where X, u are row and column indices of the matrix, respectively. However, the 
computational algorithms used here for the calculation of Markovian texture features 
uses so-called sum and difference histograms: H\ and H'j , where is the 
probability of neighboring pixels having grey levels which sum to 1, and H d m is the 
probability of neighboring pixels having grey level differences of m, where an 8- 
connected neighborhood is assumed. Values of grey levels, 1, m, used in the sum and 
difference histogram are obtained by quantization of the dynamic range of each 
individual object into 40 levels. 

For completeness, the formulae that follow for Markovian texture features 
include both the conventional formulae and the computational formulae actually used. 

V.l entropy 

The entropy feature represents a measure of "disorder" in object grey level 
organization, large values correspond to very disorganized distributions, such as a 
"salt and pepper" random field: 

entropy = ZZ A x ^ log w &X M (conventional) 
X H 

entropy = -^HJ log,. H\ -^H' m log,. H' m (computational) (50) 



V.2 energy 

The energy feature gives large values for an object with a spatially organized 
grey scale distribution. It is the opposite of entropy, giving large values to an object 
with large regions of constant grey level: 

energy = 11 A 2 , (conventional) 
X |i ,K 



energy = I(//, J ) + I(ff£) 2 (computational) (51) 
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V.3 contrast 

The contrast feature gives large values for an object with frequent large grey 
scale variations: 

contrast = II(^-n) 2 Axn {conventional) 



contrast = I m 1 Hf n {computational) (52) 



•m 

m 



V.4 correlation 

A large value for correlation indicates an object with large connected 
subcomponents of constant grey level and with large grey level differences between 
adjacent components. 

10 correlation = II (X - l q )(u -I q )&\, u {conventional) 

correlation = -f I (/ - 2l q )H s , - I m 2 H^) {computational) (53 ) 

where T' is the mean intensity of the object calculated for the grey scale quantized to 
40 levels. 

V.5 homogeneity 

1 5 The homogeneity feature is large for objects with slight and spatially smooth 

grey level variations: 

homogeneity = I I r A X fl {conventional) 

\ u i + (X-nr 

• homogeneity = Y ^-—H^ {computational) (54) 

„0 + »0 
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V.6 cl_shade 

The cl_shade feature gives large absolute values for objects with a few distinct 
clumps of uniform intensity having large contrast with the rest of the object. Negative 
values correspond to dark clumps against a light background while positive values 
5 indicate light clumps against a dark background: 



cl_ shade = H(X + fi - 2/4) 3 A x ,n {conventional) 

x n 



cl shade 



(computational) 



(55) 




10 



V.7 



cl_promincncc 

The feature cl_prominence measures the darkness of clusters. 

cl_ prominence = £ £ (X + n - 21 q ) 4 A X yL (conventional) 



cl_ prominence = 




(computational) 



(56) 
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VI Non-Markovian Texture Features 

These features describe texture in terms of global estimation of grey level 
differences of the object. 

VI. 1 denjitspot, dendrkspot 

These are the numbers of local maxima and local minima, respectively, of the 
object intensity function based on the image averaged by a 3 x 3 window, and divided 
by the object area. 

I K4 

c max 

/'=i /'=i >J 

den_ lit spot = (57) 

A 

and 

L M . 

I 0 jt j, 

den_ drk_ spot = ' ~ ly ' =1 (58) 



where 



and 



17 if there exists a local maximum of ]U r with value max r r 
g max „ I J * J ,J 

l ' jt \0 otherwise 



[l if there exists a local minimum of //» with value min-.* ,< 

6 "7,= J J J J 

,J \0 otherwise 



and where 



and / is the object intensity, Q is the object mask, and A is the object area. 
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VL2 range_extreme 

This is the intensity difference between the largest local maximum and the 
smallest local minimum of the object intensity function, normalized against the slide 
DNA amount, iod norTn > defined in Section III. 2. The local maxima, max., y and 

minima, min lV , are those in Section VI. 1 above. 



range_ extreme = (max(maxj> j> ) - (w/H(/n//i /( y )) 



100 



iod 



norm * 



(59) 



10 



VT.3 range_avcrage 

This is the intensity difference between the average intensity of the local 
maxima and the average intensity of the local minima, normalized against the slide 
DNA amount value, iodnorm, defined in Section III. 2 above. The local maxima, 
max,. .. and minima, min,., y , values used are those from Section VI. 1 above. 



range_ average - 



( L M L M ^ 

Z Z max v y Z E min \\j 
/'=1 /=!, ' /'=l/'-l 



L M 



L M 



^ ^ c max -c- c mm 

Z Z 5,.- Z Z o r j . 

I' = l/'=1 J /*=| y' = l ' J J 



100 



iod 



(60) 



norm 



VI.4 center_of_gravity 

The center^ofjjravity feature represents the distance from the geometrical 
15 center of the object to the "center of mass" of the optical density function, normalized 
by the mean_radius of the object: 



center_ of_ gravity - 



f L M 
ZZi'ODijtlij 

f=lj/ " 1 x ceniroid 

L M 

Z Z ODijSlfj 



f L M 
ZTJ-ODijClij 

/=iy=i 

L M 

Z Z ODijSlij 



- y cemroid 



mean radius 



(61) 



This gives a measure of the nonuniformity of the OD distribution. 
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VII Fractal Texture Features 

" The fractal texture features are based on the area of the three-dimensional 
surface of the object's optical density represented essentially as a three-dimensional 
bar graph, with the vertical axis representing optical density, and the horizontal axes 

S representing the x and v spatial coordinates. Thus, each pixel is assigned a unit area in 
the .v - y plane plus the area of the sides of the three-dimensional structure 
proportional to the change in the pixel optical density with respect to its neighbors. 
The largest values of fractal areas correspond to large objects containing small 
subcomponents with high optical density variations between them. 

10 The difference between fractal l_area and fractal2_area is that these features 

are calculated on different scales: the second one is based on an image in which four 
pixels are averaged into a single pixel, thereby representing a change of scale of 
fractall_area. This calculation needs the additional mask transformation: 
f2, 2 p represents the original mask fi with 4 pixels mapped into one pixel and any 

15 square of 4 pixels not completely consisting of object pixels is set to zero. Clij' 
represents Cl i2J7 expanded by 4 so that each pixel in Cl tljl is 4 pixels in Clij . 

VH.l fractal l_area 

fractal 1_ area = £ I {OD-j-OD*^ + OD* j - OA-l,,| + OP/j (M) 
i=2 ;=2 

where OD\ is the optical density function of the image scaled by a factor common to 
20 all images such that the possible optical density values span 256 levels. 

Vil.2 fractal2_area 

This is another fractal dimension, but based on an image in which four pixel 
squares are averaged into single pixels, thereby representing a change of scale of 
fractal l_area in Section VII. 1 above. 

25 fractal2_area = £ X(jOD,V,: - OD* y) . , | + \OD' t h - OD' 7 _ lh | + 1)Q () ;) (63) 

',=2^=2 

with L-2, M2 as integers, and OD' i2 j7 is a scaled 



where, L*i - 



Ml 



2 



optical density function of the image, with 4 pixels averaged into one. 
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VD.3 fractal_dimen 

The fractal_dimen feature is calculated as the difference between logarithms of 
fractall_area and fracla!2_area, divided by log 2. This varies from 2 to 3 and gives a 
measure of the "fractal behavior" of the image, associated with a rate at which 
5 measured surface area increases at finer and finer scales. 

\0B, w (Jractal\_area) - \og l 0 (Jractal2_area) 

fractal dimen = ; — 

login 2 

VIII Run Length Texture Features 

Run length features describe texture in terms of grey level runs, representing 
sets of consecutive, collinear pixels having the same grey level value. The length of 
10 the run is the number of pixels in the run. These features are calculated over the image 
with intensity function values transformed into 8 levels. 

The run length texture features are defined using grey level length matrices, 
<R® for each of the four principal directions: 9 = 0°, 45°, 90°, 135°, where the 
directions are defined clockwise with respect to the positive x-axis. Note: As defined 
15 here, the run length texture features are not rotationally invariant, and therefore, 
cannot, in general, be used separately since for most samples there will be no a priori 
preferred direction for texture. For example, for one cell, a run length feature may be 
oriented at 45°, but at 90° in the next; in general, these are completely equivalent. 
Each element of matrix K% q specifies the number of times that the object contains a 
20 run of length q, in a given direction, 0, consisting of pixels lying in grey level range, p 
(out of 8 grey levels). Let N s = 8 be the number of grey levels, and N r be the number 
of different run lengths that occur in the object; then the run length features are 
described as follows: 

VTTI.l short0_runs, short45_runs, short90_runs, shortl35_runs 

25 These give large values for objects in which short runs, oriented at 0°, 45°, 

90°, or 135°, dominate. 

H^v (65) 

Z Z n% 

p=lq=\ 



WO 97/43732 



-3 5*- 



PCT/CA97/00301 



VDI.2 Iong0_runs, long45_runs, long90_runs, longl35_runs 

These give large values for objects in which long runs, oriented at 0°, 45°, 90°, 
or 135°, dominate. 



long9_ ains - 



I Z q 2 K% 



(66) 



5 VEUL3 grcyOJcvcl, grey45Jeve!, grcy90_levcl, greyI35Jcvel 

These features estimate grey level nonuniformity, taking on their lowest values 
when runs are equally distributed throughout the grey levels. 

\1 



N g 



N r 
Z » 



grey G_ level = 



P=\\q=\ 



e 
P.<7 



J 



p=l<7=l 



(67) 



VIII.4 runOJength, run45Jength, run90Jcngth, runl35_length 

10 These features estimate the nonuniformity of the run lengths, taking on their 

lowest values when the runs are equally distributed throughout the lengths. 



run 9_ length : 



N r 

z 
9=1 



Z * 



e 
pa 



Z Z *% q 

p=\q=\ 



(68) 
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Vm.5 runO_percent, run45_percent, run90_percent, runl35_percent 

These features are calculated as the ratio of the total number of possible runs 
to the object's area, having its lowest value for pictures with the most linear structure. 



run 8_ percent = 



P= 1 ^ 1 (69) 



5 where A is the objects area. 



V1EL6 texture_orient 

This feature estimates the dominant orientation of the object's linear texture. 



texture_ orient = 



180 



7t 



It 

— + arcian 
2 



(X\ — y pseudo- m oment 2 ) 
^ pseudo-cross ^moment 2 



(70) 



where X\ is the maximal eigenvalue of the run length pseudo-second moment matrix 
,0 (calculated in analogy to Section 11.9). The run length pseudo-second moments are 
calculated as follows: 



Xpscudo - momcnt2 = ^^^^ 



II 

ypacudo - momcntZ = 



(71) 
(72) 



xypseudo - cross _ momenl2 - 



A/* N r 

- r i 



• zr2/ 2 - V2/; 



/=i 



2J2 



15 



(73) 



Orientation is defined as it is for cell_orient, Section 11.9, as the angle 
(measured in a clockwise sense) between the y axis and the dominant onentat.on of 

the image's linear structure. 
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VHL7 size_txt_orient 

This feature amplifies the texture orientation for long runs. 

X' 

size txt orient = -- L (74) 

- - a; 

where X\,X\ are the maximal and minimal eigenvalues of the runjength pseudo- 
second moment matrix, defined in Section VIII.6. 

Each of the above features are calculated for each in-focus object located in 
the image. Certain features are used by the classifier to separate artifacts from cell 
nuclei and to distinguish cells exhibiting MACs from normal cells. As indicated 
above, it is not possible to predict which features will be used to distinguish artifacts 
from cells or MAC cells from non-MAC cells, until the classifier has been completely 
trained and produces a binary decision tree or linear discriminant function. 

In the present embodiment of the invention, it has been determined that 
thirty (30) of the above-described features appear more significant in separating 
artifacts from genuine nuclei and identifying cells with MACs. These primarily 
texture features are as follows: 



1) Area 

2) mean radius 

3) OD variance 

4) OD skewness 

5) range average 

6) OD maximum 

7) density of light spots 

8) low DNA area 

9) high DNA area 

10) low DNA amount 



30 preferred nuclear features 

11) high DNA amount 

12) high average distance 

13) mid/high average distance 

14) correlation 

15) homogeneity 

16) entropy 

17) fractal dimension 

18) DNA index 

19) run 0 percent 

20) run 45 percent 



2 1 ) run 90 percent 

22) run 135 percent 

23) grey level 0 

24) grey level 45 

25) grey level 90 
25) grey level 135 

27) run length 0 

28) run length 45 

29) run. length 90 

30) run length 135 



Although these features have been found to have the best ability to 
differentiate between types of cells, other object types may be differentiated by the 
other features described above. 

As indicated above, the ability of the system according to the present invention 
to distinguish cell nuclei from artifacts or cells that exhibit MACs from those that do 
not depends on the ability of the classifier to make distinctions based on the values of 
the features computed. For example, to separate cell nuclei from artifacts, the present 
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invention may apply several different discriminant functions each of which .. train* 
t0 identify particular types of objects. For example, the followmg d.scnm.nant 
function has been used in the presently preferred embodiment of the invenuon to 
separate intermediate cervical cells from small picnot.c objects: 



max_radius 

freq_low_fft 

harmon03_fft 

harmon04.ff\ 

lowVSmed_DNA 
energy 
correlation 
homogeneity 
cl_shade 
den_drk_spot 
CONSTANT 



cervical cells 
4.56914 
-.03624 
1.29958 
.85959 
58.83394 
6566.14355 
.56801 
-920.05017 
-67.37746 
916.69360 
-292.92908 



picnotic 

3.92899 

-.04714 

1.80412 

1.20653 

61.84034 

6182.17139 

.52911 

-883.31567 

-63.68423 

870.75739 

-269.42419 



Another discriminant function that can separate cells from junk part.cles ,s: 



eccentricity 
compactness 
freq__low_fft 

freq_high_fft 
harmon02.fft 
harmon03 .fft 
medDNAamnt 
correlation 

CONSTANT 

Yet a third discriminant function that can separate folded cells that should be 

ignored from suitable cells for analysis, 



cells 


junk 


606.67365 


574.82507 


988.57196 


1013.19745 


-2.57094 


-2.51594 


-28.93165 


-28.48727 


-31.30210 


-30.18383 


14.40738 


14.30784 


39.28350 


37.50647 


.27381 


.29397 


-834.57800 


-836.19659 



sphericity 
eccentricity 
compactness 
elongation 



normal interm 

709.66357 

456.09146 

1221.73840 

-391.76352 



rejected objects 

701.85864 
444.18469 

1232.27441 

-387.19376 
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freq_high_f¥t 

lowDNAamnt 

low_den_obj 

correlation 

range_average 

CONSTANT 



-37.89624 -37.39510 
-41.89951 -39.42714 
1.40092 1.60374 
.26310 .29536 
.06601 .06029 
-968.73628 -971.18219 
Obviously, the particular linear discriminant function produced by the classifier 
will depend on the type of classifier used and the training sets of cells, The above 
examples are given merely for purposes of illustration. 

As can be seen, the present invention is a system that automatically detects 
malignancy-associated changes in a cell sample. By properly staining and imaging a 
cell sample, the features of each object found on the slide can be determined and used 
to provide an indication whether the patient from which the cell sample was obtained 
is normal or abnormal. In addition, MACs provide an indication of whether cancer 
treatment given is effective as well as if a cancer is in remission. 

While the preferred embodiment of the invention has been illustrated and 
described, it will be appreciated that various changes can be made therein without 
departing from the spirit and scope of the invention. 
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The embodiments of the invention in which an exclusive property or privilege 
is claimed are defined as follows: 

I. A method of detecting malignancy-associated changes in a cell sample, 

comprising the steps of: 

. obtaining a cell sample; 

staining the sample to identify cell nuclei within the sample; 
obtaining an ima g e of the cell sample with a digital microscope of the type that 
includes a digital CCD camera and a programmable slide stage; 
focusing the image; 
identifying objects in the image; 
calculating a set of feature values for each object; and 

analyzing the feature values to determine whether each object is a cell nucleus 
having malignancy-associated changes. 

2 The method of Claim 1, wherein each object in the cell image 
comprises a group of pixels each of which has an intensity value, wherein the step of 
identifying objects in the image further comprises the steps of: 
computing a histogram of all pixels in the image; 
determining an average intensity of background pixels in the image; 
determining an average intensity of the object pixels in the image; 
determining a threshold that lies between the average intens.ty of the 
background pixels and the average intensity of the object pixels; and 

identifying every pixel with an intensity greater than the threshold as 
background and every pixel with an intensity less than or equal to the threshold as an 
object. 

3 The method of Claim I , wherein each of the objects has an edge that 
separates the object from the background and wherein the step of identifying objects 
in the cell image further comprises the steps of. 

identifying an edge of the object by performing the steps of. 

. creating an annular ring that surrounds the object by dilating the edge 
of the object to define an outer edge of the annular ring and eroding the edge of the 
object to define an inner edge of the annular ring; 

calculating a gradient value of each pixel in the annular ring; and 
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removing pixels from the annular ring having lower gradients until the 
annular ring comprises a single pixel chain that encircles the object. 

4. The method of Claim 3, wherein the step. of calculating one or more 
features of the set of feature values further comprises the step of: 

dilating or contracting the true edge of the object before calculating the 
features. 

5. The method of Claim 1, wherein the image is stored in a memory of a 
computer system, the method further comprising the steps of: 

adjusting the local focus of each object by obtaining a series of images at 

different stage positions; 

for each object, selecting the image where the focus of the object is best; and 
overwriting the memory of the computer with the image of each object at its 

best focus. 

6. The method of Claim 5, further comprising the step of: 
compensating the image for variations in illumination intensity of a light source 

of the digital microscope by : 

reading a test image obtained from a blank slide in the digital 

microscope; 

subtracting the intensity value of each pixel of the test image from a 
corresponding pixel in the image of the cell sample; 

determining the average intensity of the pixels in the test image; and 
adding the average intensity to each pixel of the image obtained from the cell 

sample. 

7. The method of Claim 6, further comprising the step of compensating 
for local absorbency around the object by: 

determining an average pixel intensity for a group of pixels near the object; 

and 

subtracting the average pixel intensity for the group of pixels near the object 
from each pixel that is included in the object. 

8. The method of Claim 7, wherein the group of pixels near the object are 
defined by a square boundary having an area slightly greater than an area of the 
object. 
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9. The method of Claim 1, further comprising the steps of: 

removing artifacts from the image by calculating the area, shape and optical 
density for each object; and 

removing from the image any object with an area > 2,000 square microns, any 
objects with a shape or sphericity greater than 4 and an optical density greater than 
lc 

10. The method of Claim 1, wherein the features used to separate cell 
nuclei having malignancy-associated changes from nuclei not having malignancy- 
associated changes arc selected from the group comprising: 



1) Area 

2) mean radius 

3) OD variance 

4) OD skewness 

5) range average 

6) OD maximum 

7) density of light spots 

8) low DNA area 

9) high DNA area 

1 0) low DNA amount 



11) high DNA amount 

12) high average distance 

13) mid/high average distance 

1 4) correlation 

15) homogeneity 

16) entropy 

17) fractal dimension 

18) DNA index 

19) run 0 percent 

20) run 45 percent 



21) run 90 percent 

22) run 135 percent 

23) grey level 0 

24) grey level 45 

25) grey level 90 
25) grey level 135 

27) run length 0 

28) run length 45 

29) run length 90 

30) run length 135 



11. A method of predicting whether a patient will develop cancer, 
comprising the steps of: 

obtaining a sample of apparently normal cells from the patient; 

determining whether the cells in the sample exhibit malignancy associated 
changes by: 

(1) staining the nuclei of the cells in the sample; 

(2) obtaining an image of the cells with a digital microscope and 
recording the image in a computer system; 

(3) analyzing the stored image of the cells to identify the nuclei; 

(4) computing a set of feature values for each nucleus found in the 
sample and from the feature values determining whether the nucleus exhibits a 
malignancy associated change; and 

determining a total number of nuclei in the sample that exhibit malignancy- 
associated changes and from the number predicting whether the patient will develop 
cancer. 



WO 97/43732 



PCT/CA97/00301 



1/12 



12 < 



14 



DIGITAL HIGH 
RESOLUTION 
CCD CAMERA 

i nmuiiliii trrrm — 



DATA 



^32 

DIGITAL IMAGE 
PROCESSING 
UNIT 



■36 




7S 



». 


IMAGE 
MONITOR 


»- 









^40 



COMPUTER BUS 



34 



1 



20 



MOTORIZED 
COMPUTER CONTROLLED 
PRECISION STAGE WITH 
SLIDE LOADING UNIT 



COMPUTER 
MONITOR 



V 



COMPUTER 



V 



KEYBOARD 
MOUSE 



38 



30 



PERIPHERALS 



1 



42 



10- 



SUBSTITUTE SHEET (RULF ?fi^ 



WO 97/43732 



PCT/CA97/003O1 



2/1 2 



OBTAIN SAMPLE 



Lf 



50 



1 \f 52 



STAIN 



SCAN PORTION OF 
SLIDE INTO MEMORY 



f 



54 




MOVE STAGE TO 
NEW POSITION 



f 



60 



FIELD \JES_ 
BLANK 

7 



GLOBAL 
FOCUS 



62 



HISTOGRAM 
ALL PIXELS 



S 



64 




RIIRRTITIJTF RHFFT mill F 9fi\ 



WO 97/43732 



PCT/CA97/00301 



3/12 - 






r 




SET THRESHOLD GREY 
LEVEL FROM HISTOGRAM 


f 








IDENTIFY ALL 
PIXELS WITH 
INTENSITY< THRESHOLD 





68 



72 



CALCULATE AREA, SHAPE, 
OPTICAL DENSITY 
OF EACH OBJECT 



1/ 



74 



REMOVE FROM MEMORY 
OBJECTS THAT CANNOT 
BE CELL NUCLEI 



76 




n inrrin itt oucCT fO\\\ C OC\ 



WO 97/43732 



PCT/CA9 7/00301 




COMPENSATE FOR 
VARIATIONS IN 
ILLUMINATION 



80 



OBTAIN LOCAL 
FOCUS FOR OBJECT 




82 



ANY 

IN-FOCUS \ NO 
OBJECTS FOUND. 



COMPENSATE FOR 
LOCAL ABSORBANCE 
AROUND OBJECT 



85 



DETERMINE EDGE OF 
EACH IN-FOCUS OBJECT 



if 



86 



■87 



CALCULATE FEATURES 



4/12 




RUN OBJECT CLASSIFIER 
ON FEATURES 



88 



SAVE IMAGE, FEATURE 
CALCULATIONS AND 
RESULTS OF CLASSIFIER FOR 
EACH IN-FOCUS OBJECT 



if 



91 




SUBSTITUTE SHEET (RULE 26) 



PCT/CA97/00301 

WO 97/43732 



5/1 2 




SUBSTITUTE SHEET (RULE 26) 



PCT/CA97/00301 

WO 97/43732 



6/1 2 






L 






r 
















CO 


1 










Q fefc cm 






CO 








g 















CM 



CO Sfc — 




SUBSTITUTE SHEET (RULE ?ffl 



WO 97/43732 



PCT/CA97/00301 




WO 97/43732 



PCT/CA97/OO301 




SUBSTITUTE SHEET (RULE 26) 



WO 97/43732 



PCT7CA97/00301 



9/12 




242-, 244~ t 







f/A 


■ 


■ 






1 









-250 



-252 



cuirstitiitf rhfft mm p or\ 



WO 97/43732 



PCT/CA97/00301 



10/12 



275 



FEATURES FROM 
UNKNOWN 
NON-NUCLEI 



■285 



FEATURES FROM 
KNOWN 
NUCLEI 



Si 



290 



CLASSIFIER TO 
SEPARATE NUCLEI 
FROM NON-NUCLEI 



Si 



294 



INDICATION OF 
WHETHER OBJECT IS A 
NUCLEUS OR AN ARTIFACT 



Si 



292 



FEATURES FROM 
UNKNOWN OBJECTS 



SUBSTITUTE SHEET (RULE 261 



WO 97/43732 



PCT/CA97/00301 



11/12- 



302 



FEATURES FROM 

NUCLEI IN 
NORMAL PATIENTS 



■304 



FEATURES FROM 
NUCLEI IN 
ABNORMAL PATIENTS 



300 



CLASSIFIER TO 
IDENTIFY NUCLEI THAT 
EXHIBIT MALIGNANCY 
ASSOCIATED CHANGES 



308 



INDICATION OF 
WHETHER NUCLEI IS 
MAC-POSITIVE OR NOT 



306 



FEATURE VALUES 

FROM 
UNKNOWN NUCLEI 



SUBSTITUTE SHEET (RULF m 



WO 97/43732 



PCT/CA9 7/00301 



12/12- 



FOR EACH NUCLEUS 
ON SLIDE FOR 
WHICH FEATURES HAVE 
BEEN CALCULATED 



:338 



325 



RUN CLASSIFIER 
ON FEATURES 



GET NEXT 


NO 


OBJECT 





330 




-332 
MAC "\ YES 
POSITIVE 

9 




OF-- 340 
'MAC-POSITIVE^ 
NUCLEI EXCEED 
^THRESHOLD, 

9 



1 



334 



INCREASE COUNT 
OF MACS 



YES 



PATIENT MAY 
HAVE CANCER 



f 



344 



■J42 



NO 


PATIENT 




HEALTHY 



SUBSTITUTE SHEET (RULE 26) 



INTERNATIONAL SEARCH REPORT 



Inte jnaJ Application No 

PCT/CA 97/00301 



A CLASSIFICATION OF SUUJECT MATTER 

IPC 6 G06K9/00 G01N15/14 



Accurd.ng in Inu-maponal latent Clavuficaonn (IPO or to both national claxsihcauon and IPC 



B. MELDS SEARCHED 



Minimum documentation searched iclawhcalmn .%y«cm ("Unwed hy classihcaoon symbols) 

IPC 6 G06K G01N G06T 



Ddcumcnuunn searched other ih.in minimum documcnuuon w> ihe extent that such documents arc included ,n the i,el* searched 



Klcclroiut daia base consulted during the imemaU.-n.il search (name <>< data hase And. where practical, search terms u«d> 



C. DOCUMKSTS CONSIDERED TO HE RELEVANT 



Category * 



Cilaonn of document, with indication, where appropriate, of the relevant parage* 



Relevant to claim Nd. 



WO 96 Q9594 A (NEOPATH INC) 28 March 1996 
see page 1 - page 14; claim 21 

WO 96 09605 A (NEOPATH INC) 28 March 1996 
see page 1 - page 11 

EP 0 595 506 A (XILLIX TECHNOLOGIES CORP) 
4 May 1994 

cited in the application 
see the whole document 

W0 93 16442 A (NEOPATH INC) 19 August 1993 
see page 1 - page 9, line 25; figure 4A 

-/- 



1,11 
1.11 
1,2,11 

1,2,11 



0 



Further document* arc listed in the continuation of hux C. 



Patent family members are listed in annex. 



1 Special categories of cited documents : 

'A" document defining the general state of the an which is not 

considered to he of particular relevance 
"E" earlier document but published on or after the international 

filing date 

*L* document which may throw doubts on pnonly clajmfs) or 
which is cued to establish the puhhcstion dale of another 
citation or other special reason (as specified) 

*()" document referring to an oral disclosure, use, exhibition or 
other means 

"P" document published pnor to the international filing dale but 
later than the priority dale claimed 



*T* later document published after the international filing dale 
or pnonry dale and not in conflict with the application hut 
cited to understand the principle or theory underlying the 
invention 

'X* document of particular relevance; the daimed invenunn 
cannot be considered novel or cannot be considered to 
involve an inventive step when Ihe document is taken alone 

'Y* document of particular relevance; the claimed invenuon 
cannot he coasidered to involve an invenove step when the 
document ts cumhined with one or more other such docu- 
ments, such comhinauon being obvious to a person skilled 
in the an. 

document member of the same patent family 



Dale of the actual completion of the inlernaunnal search 



23 July 1997 



Date of mailing of Ihe international search report 

- 6. 08. 97 



Name and mailing address of the ISA 

European Patent Office, P.B. 5818 PalcnUaan 2 
NL ■ 2280 HV Rijswijk 
Tel. ( * 3t -70) 340-2040, Tx. 31 651 cpo nJ, 
Fax t - 31 -?U) 340-3016 



Authorized officer 



Brison, 0 



Form PCT ISA'310 (Jftond !hcel| (July 1972) 



INTERNATIONAL SEARCH REPORT 

c ( t. on unuau.,n) DOCU MCNTS CHNSIUERBD TO III: RbL.KVA.N 1 

Ouoon ol document. w,<h .ndKionn. where appropriate, o. *e ,.le««nl pMU |« 



Inu Jnal Application No 

PCT/CA 97/GG3G1 



Uaicgory 



X 
A 



traC, TRENDS IN ANALYTICAL CHEMISTRY, 

vol. 10, no. 8, 1 September 1991, 

AMSTERDAM, NL, 

Daqes 237-243, XP00Q219625 

BERT RAND D ET AL: "BASICS OF VIDEO IMAGE 

ANALYSIS" 

see page 239 - page 240 

EP 0 610 916 A (CEDARS SINAI MEDICAL 
CENTER) 17 August 1994 
see the whole document 

WO 93 16436 A (NEOPATH INC) 19 August 1993 
see claims 1,2 

WO 90 10277 A (CELL ANALYSIS SYSTEMS INC) 

7 September 1990 

see page 11 - page 14, line 30 

W0 91 15826 A (NEUROMEDICAL SYSTEMS INC) 
17 October 1991 

see page 6, line 15 - page 10, line 26 



Kclcvant in claim No. 



1,3,11 



1,5,11 



^•r ,r * -mo rtf «pmnd ih.ell (JuN 1993) 



INTERNATIONAL SEARCH REPORT 

information on paicnl (anruly members 



Patent document 
cued in search report 



Publication 
date 



(nie mal Apphcation No 

PCT/CA 97/00301 



Patent family 
mcmbcr(s) 



Publication 
date 



WO 9609594 A 



28-03-96 



All ■ 3629295 A 



09-04-96 



W0 9609605 


A 


28-03-96 


AU 


3629795 A 


uy " \jh -y o 


EP 0595506 


A 


04-05-94 


CA 


2086785 A 


15-04-94 






JP 


6231229 A 


19-08-94 


W0 9316442 


A 


19-08-93 


AU 


671984 B 


i n no o c 

ly-uy-yb 






AU 


3737693 A 


03-09-93 








AU 


7545096 A 


20-02-97 








CA 


2130340 A 


19-08-93 








EP 


0664038 A 


26-07-95 








JP 


7504056 T 


27-04-95 








US 


5528703 A 


18-06-96 


EP 0610916 


A 


17-08-94 


NONE 




W0 9316436 


A 


19-08-93 


AU 


670938 B 


08-08-96 






AU 


3722893 A 


03-09-93 








CA 


2130338 A 


19-08-93 








EP 


0628186 A 


14-12-94 








JP 


7504283 T 


11-05-95 


W0 9010277 


A 


07-09-90 


CA 


2045614 A 


25-08-90 








EP 


0460071 A 


11-12-91 


W0 9115826 


A 


. 17-10-91 


AU 


7668191 A 


30-10-91 






CA 


2064571 A 


01-10-91 








EP 


0479977 A 


15-04-92 








US 


5544650 A 


13-08-96 



Form PCT ISA/310 (pauni fimtly inntt) (July 1992] 



